Reciprocal Excitation Between Biological and Robotic Research

While biological principles have inspired researchers in computational and engineering research for a long time, there is still rather limited knowledge flow back from computational to biological domains. This paper presents examples of our work where research on anthropomorphic robots lead us to new insights into explaining biological movement phenomena, starting from behavioral studies up to brain imaging studies. Our research over the past years has focused on principles of trajectory formation with nonlinear dynamical systems, on learning internal models for nonlinear control, and on advanced topics like imitation learning. The formal and empirical analyses of the kinematics and dynamics of movements systems and the tasks that they need to perform lead us to suggest principles of motor control that later on we found surprisingly related to human behavior and even brain activity. INTRODUCTION When searching for a general framework of how to formalize the learning of coordinated movement, some of the ideas developed in the middle of the 20th century still remain useful. At this time, theories from optimization theory, in particular in the context of dynamic programming (Bellman, 1957; Dyer & McReynold, 1970), described the goal of learning control in learning a policy. A policy is formalized as a function that maps the continuous state vector x of a control system and its environment, possibly in a time dependent way, to a continuous control vector u: u x = π( ) , , α t (1) The parameter vector α denotes the problem specific adjustable parameters in the policy !—not unlike the parameters in neural network learning. At the first glance, one might suspect that not much was gained by this overly general formulation. However, given some cost criterion that can evaluate the quality of an action u in a particular state x, dynamic programming, and especially its modern relative, reinforcement learning, provide a well founded set of algorithms of how to compute the policy ! for complex nonlinear control problems. Unfortunately, as already noted in Bellman’s original work, learning of ! becomes computationally intractable for even moderately high dimensional state-action spaces. Although recent developments in reinforcement learning increased the range of complexity that can be dealt with (e.g., [1]; [2]; [3]), it still seems that there is a long if not impossible way to go to apply general policy learning to complex control problems. In most robotics applications, the full complexity of learning a control policy is strongly reduced by providing prior information about the policy. The most common priors are in terms of a desired trajectory, [ ( ), ̇ ( )] x x d d t t , usually hand-crafted by the insights of a human expert. For instance, by using a PD controller, a (explicitly time dependent) control policy can be written as: